Skip to content

feat(plugin_repo): add trialware check#1319

Open
faisalahammad wants to merge 3 commits into
WordPress:trunkfrom
faisalahammad:feature/1313-trialware-check
Open

feat(plugin_repo): add trialware check#1319
faisalahammad wants to merge 3 commits into
WordPress:trunkfrom
faisalahammad:feature/1313-trialware-check

Conversation

@faisalahammad

@faisalahammad faisalahammad commented May 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a new Trialware_Check under the plugin_repo category to detect potential trialware and locked built-in functionality in plugins submitted to WordPress.org.

The check scans PHP and JS files for five categories of gating patterns: license key gates, pro/premium plan gates, trial period gates, usage quota limits, and payment/subscription gates.

Fixes #1313

Changes

includes/Checker/Checks/Plugin_Repo/Trialware_Check.php

New check class extending Abstract_File_Check. Defines PATTERN_GROUPS constant with five pattern groups, each with regex patterns, result code, and description message. Deduplicates per-file matches within each group to avoid noise.

Result codes:

  • trialware_license_gate_candidate
  • trialware_pro_premium_gate_candidate
  • trialware_trial_gate_candidate
  • trialware_quota_gate_candidate
  • trialware_payment_gate_candidate

includes/Checker/Default_Check_Repository.php

Registered trialware slug mapping to Trialware_Check instance.

docs/checks.md

Added trialware row to the available checks table.

Response to review (davidperezgar)

Reviewer asked for candidate-only detection with an AI confirm/discard step, matching issue #1313's acceptance criteria, rather than reporting every regex match straight as an error.

Investigation found plugin-check already has this exact mechanism, generically, not per-check: AI_Analyzer::get_ai_prompt_map() (includes/Traits/AI_Analyzer.php) maps check-code prefixes to a prompt file, and Abstract_Check_Runner::run() runs analyze_results_with_ai() after all checks finish (opt-in via CLI --ai flag / admin AI toggle), tagging each issue is_false_positive + explanation. The CLI (Plugin_Check_Command::split_false_positive_results()) and Admin AJAX already split confirmed vs. false-positive for display. Every AI-reviewed check today (EscapeOutput, NonceVerification, DirectDatabaseQuery, Obfuscation, SettingSanitization, PluginUpdater) uses this same generic pass — none of them call AI to decide error vs. discard inside the check class itself.

Rather than duplicating that infrastructure with a new bespoke per-check AI workflow, this update wires trialware into the existing generic pass:

  • includes/Traits/AI_Analyzer.php — added trialware_ prefix to get_ai_prompt_map(), pointing at a new prompt file. All five trialware_*_candidate codes match this prefix.
  • prompts/ai-review-trialware.md (new) — prompt defining genuine locked-functionality gating vs. false positive, explicitly covering the review's examples: external service API keys, a separate premium plugin/product, generic marketing copy, and harmless license/update-checker wording.
  • Trialware_Check.php — broadened PATTERN_GROUPS per the review's examples: activation codes, "free trial ended/expired" wording, "to unlock ... feature" / "limit reached" phrasing, and lite/free-version gating. Kept the existing five _candidate result codes. Added a class docblock explaining the AI-candidate relationship so it's clear this check intentionally does not call the AI client itself.
  • Tests — added three false-positive-shaped functions to the clean fixture (external API key check, mention of a separate premium plugin, EDD-style update-checker license) to prove the regex layer itself doesn't over-match before AI ever runs.

This keeps candidates flagged the same way as every other AI-reviewed check, without a second, incompatible AI-review code path.

Testing

Test 1: Plugin with trialware patterns (should flag errors)

Fixture: test-plugin-trialware-with-errors contains is_licensed(), is_pro(), trial_expired(), quota_exceeded(), and has_paid() calls.

Expected: 5 errors reported, one per result code.

Test 2: Plugin without trialware patterns (should pass clean)

Fixture: test-plugin-trialware-without-errors contains standard admin page, settings save, textdomain loading, plus the new false-positive-shaped functions added in response to review.

Expected: 0 errors, 0 warnings.

Test 3: Quality checks

composer lint     ✅
composer phpstan  ✅ (0 errors)
composer phpmd    ✅
npm run test-php  ✅ (485 tests, 1409 assertions)
coderabbit review --agent ✅ (0 findings)

Note: the branch had fallen behind trunk (mergeable_state: dirty); merged trunk in as part of this update to pick up AI_Analyzer and resolve two trivial append-only conflicts in docs/checks.md and Default_Check_Repository.php.

AI Usage Disclosure

  • AI-assisted

Claude Code was used to: investigate the existing AI-review architecture (AI_Analyzer, Abstract_Check_Runner, CLI false-positive splitting) referenced above, merge trunk and resolve conflicts, wire the trialware_ prefix into the prompt map, draft prompts/ai-review-trialware.md, broaden the regex patterns per review feedback, and add the false-positive test fixtures.

Open WordPress Playground Preview

Add new Trialware_Check under plugin_repo category to detect
trialware and locked built-in functionality in plugin submissions.

Scans PHP and JS files for five pattern groups:
- License key gates
- Pro/premium plan gates
- Trial period expiration gates
- Usage quota limits
- Payment/subscription gates

Each match reported as error with dedicated result code
(trialware_{type}_candidate). Patterns designed to minimize
false positives on legitimate external service integrations.

Fixes WordPress#1313
@github-actions

github-actions Bot commented May 25, 2026

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: faisalahammad <faisalahammad@git.wordpress.org>
Co-authored-by: marceltannich <mardroid@git.wordpress.org>
Co-authored-by: davidperezgar <davidperez@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@marceltannich

Copy link
Copy Markdown
Member

Tested during WCEU 2026 Contributor Day.

Manual testing:

  • Tested the functionality introduced in this PR (via the 2 test plugins and some others I had available)
  • Behavior matched expectations.

Automated checks:

  • ✅ composer lint
  • ✅ composer phpstan
  • ✅ composer phpmd
  • ✅ PHPUnit (477 tests, 1352 assertions)

Result:

  • All checks passed.
  • No issues found during testing.

Note:

  • PHPUnit reported a phpunit.xml validation warning (filter element not expected), but the full test suite completed successfully and all tests passed.

@davidperezgar

Copy link
Copy Markdown
Member

Thanks for working on this. I reviewed the PR and also used ChatGPT to help compare it against our internal trialware scanner. The conclusion is that this is a good start, but I don’t think it satisfies the issue yet.

The main gap is that the issue asks for an AI-assisted candidate workflow, not a direct regex-to-error check.

Right now this PR:

  • scans PHP/JS files for a few trialware-like regex patterns;
  • immediately reports every match as an error;
  • does not distinguish candidates from confirmed violations;
  • does not run an AI review step;
  • does not hide/discard false positives;
  • does not include coverage for acceptable cases like external services, separate premium plugins, or harmless license/update wording.

The internal scanner this is based on works differently:

  • it first detects broad candidate patterns such as license keys, activation codes, free trial wording, lite/free version limits, “to unlock”, “limit reached”, plan limits, etc.;
  • it sends those candidates to an AI review prompt;
  • the AI is asked to confirm whether bundled plugin functionality is actually restricted;
  • AI-confirmed cases become actionable issues;
  • AI-discarded cases are removed from the final output.

That distinction is important because trialware detection has a high false-positive risk. For example, a plugin may mention a premium version, use an external service API key, or include license/update-related wording without actually locking functionality bundled in the WordPress.org plugin. Those should not become final blocking errors.

To move this closer to the requested behavior, I think this needs:

  1. Candidate result codes, e.g. trialware_locked_feature_candidate.
  2. Confirmed result codes, e.g. trialware_locked_feature_detected.
  3. An AI prompt/check step that decides whether the candidate is a real locked built-in feature.
  4. Logic to suppress AI-discarded candidates from final results.
  5. Broader candidate patterns matching the internal scanner, including UI/readme/string-level restriction wording.
  6. Tests for both confirmed trialware and false-positive cases.

So I would not consider this complete yet. It is a useful skeleton for the static candidate detection piece, but it is missing the key AI confirmation/discard workflow from the issue acceptance criteria.

…re-check

# Conflicts:
#	docs/checks.md
#	includes/Checker/Default_Check_Repository.php
…ew pass

- add trialware_ prefix to AI_Analyzer::get_ai_prompt_map()
- add prompts/ai-review-trialware.md defining genuine gate vs false positive
- broaden Trialware_Check regex patterns (activation codes, free trial wording,
  to unlock/limit reached phrasing, lite-version gating)
- add false-positive-shaped fixtures (external API key, separate premium plugin
  mention, EDD-style license) to the clean test plugin

Addresses PR feedback from davidperezgar: candidates now flow through the same
--ai runner pass already used by EscapeOutput/NonceVerification/Obfuscation/etc,
instead of a new bespoke per-check AI workflow.

Refs WordPress#1319
@faisalahammad

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review. Good catch on the missing AI confirm/discard step.

Looked into how plugin-check already handles this for other checks. Turns out there's a generic mechanism already built for exactly this: AI_Analyzer::get_ai_prompt_map() maps check-code prefixes to a prompt file, and Abstract_Check_Runner::run() runs an AI pass after all checks finish (via --ai flag), tagging each candidate as confirmed or false-positive. EscapeOutput, NonceVerification, DirectDatabaseQuery, Obfuscation, SettingSanitization, and PluginUpdater all use this same pass today — none of them call AI inside the check class itself.

So instead of building a second, bespoke AI workflow just for trialware, I wired the trialware_*_candidate codes into that existing pass:

  • Added trialware_ to get_ai_prompt_map().
  • Added prompts/ai-review-trialware.md, which explicitly tells the AI to treat external service API keys, a separate premium plugin/product, and harmless license/update-checker wording as false positives, not confirmed gates.
  • Broadened the regex patterns to cover your examples: activation codes, "free trial ended/expired", "to unlock ... feature", "limit reached", lite/free-version gating.
  • Added 3 false-positive-shaped functions to the clean test fixture (external API key, separate premium plugin mention, EDD-style license) — still 0 errors, so the regex layer itself doesn't over-match before AI ever runs.

This keeps trialware consistent with how every other AI-reviewed check works, and the candidate/confirm/discard behavior you asked for comes from the existing runner pass rather than new plumbing. Full details in the updated PR description. Let me know if you'd rather see dedicated _detected result codes instead of relying on the existing ai_stats/ai_analysis output — happy to adjust if that's a hard requirement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Trialware Check

3 participants